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Executive Summary 

Method 

A total of twenty-two people participated in a usability study comparing two versions of 
tine Oracle Document Navigation Prototype with the DocumentationUbrary. The first 
round of tests (with eleven participants) was conducted in IBHHb "^^^ second 
round of tests (with a different set of eleven participants and aH'evisea prototype) was 
conducted in MHHB K Both rounds were conducted in a usability laboratory at Oracle 
headquartersnEacrrone-on-one session lasted about 90 minutes and was videotaped 
for later review (copies provided). 

The goal of the research was to measure user performance and opinions on a set of 
benchmark tasks, in order to identify potential usability "defects" in the prototype. 
Participants were a mix of developers, database administrators, and support personnel. 
Most had some previous experience with the Oracle Documentation Library. Users 
viewed the site with Netscape Communicator 4.7 on a 17" monitor at 1024 by 768 
pixels resolution. 

After hearing a very brief overview of test objectives, users were given a series of pre- 
defined tasks that covered the major elements of the user interface. In each round of 
testing, six selected tasks were performed using both versions of the prototype as well 
as the Documentation Library, so that users could make direct comparisons between 
tools. Participants verbalized what they were doing and thinking as they performed 
each task. A task ended either when the goal was attained or when participants said 
they would give up. At the completion of each task, users rated its difficulty on a seven- 
point scale. 

Results 

In Round 1 , the Prototype outperformed the Library in both user performance and user 
ratings. For the six common tasks, using the Library took 1 5% longer than using the 
Prototype. Users completed 100% of the tasks using the Prototype, but only 82% using 
the Library. The mean usability rating was 4.1 for the Library and 5.3 for the Prototype 
(where 1 =Very Hard and 7=Very Easy). And the mean post-session rating directly 
comparing the two tools was 6.3 (where 1=Library Much Better and 7=Prototype Much 
Better). 

For Round 2, several changes were made to make the Prototype even easier to use, by 
simplifying the presentation of its navigation tools. Again, the Prototype outperformed 
the Library in both user performance and user ratings. For the same six common tasks, 
using the Library took 28% longer than using the Prototype. Users completed 97% of 
the tasks using the prototype, but only 82% using the Library. The mean usability rating 
was 4.0 for the Library and 5.4 for the Prototype (where 1 =Very Hard and 7=Very 
Easy). And the mean post-session rating directly comparing the two tools was 6.4 
(where 1=Library Much Better and 7=Prototype Much Better). Compared to Round 1, 
more users noticed the Prototype's wide array of navigation tools, including the Virtual 
Book feature, which impressed all users in both Rounds. 
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Recommendations 

After two rounds of testing, several additional Prototype changes could improve user 
performance and acceptance even more. 

1 . Further improve the display of choices on the Prototype's home page. 

2. Clarify the relationship between the keyword field ^d the "p*-selected" list in 
the Virtual Book. ' • '* 

3. Replace full-te)<:t search checkboxes with radio buttons. 

4. Navigate directly to the first instance of the selected topic in a document and 
highlight each instance in the document. 

Methods 
Procedure 

A total of twenty-two people participated in a usability study comparing two versions of 
the Oracle Document Navigation Prototype with the Docu mentation L ibrary. The first 
round of tests (with eleven participants) was conducted in flHHB The second 
round of tests (with a different set of eleven participants anoarevised prototype) was 
conducted in MMB|B| Both rounds were conducted in a usability laboratory at Oracle 
headquarters.^acn one-on-one session lasted about 90 minutes and was videotaped 
for later review (copies provided). 

The goal of the research was to measure user performance and opinions on a set of 
benchmark tasks, in order to identify potential usability "defects" in the prototype. 
Participants were a mix of developers, database administrators, and support personnel 
recruited by User Focus, Inc. from a list of Oracle users provided by the User Interface 
Lab. Most had some previous experience with the Oracle Documentation Library. Gary 
Felfoldy of User Focus conducted all test sessions and logged user actions and 
comments. Users viewed the site with Netscape Communicator 4.7 on a 17" monitor at 
1024 by 768 pixels resolution. 

After hearing a very brief overview of test objectives, users were given a series of pre- 
defined tasks that covered the major elements of the user interface. In each round of 
testing, six selected tasks were performed using both versions of the prototype as well 
as the Documentation Library, so that users could make direct comparisons between 
tools. Participants verbalized what they were doing and thinking as they performed 
each task. A task ended either when the goal was attained or when participants said 
they would give up. At the completion of each task, users rated its difficulty on a seven- 
point scale. 
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Tasks 

Task 1 (Used for both Library & Prototype) 

You want to set a default role for a user. What command do you use? 

Task 2 (Both) 

What is the difference between PCTFREE and PCTUSED? 

Task 3 (Prototype only) 

What is the default length for a CHAR variable? What is the maximum length? 

Task 4 (Prototype only) 

You want to use the following functions in your code. Find an example of each that 
shows the correct syntax. 

a) ADD_MONTHS 

b) DBMS_LOB.TRIM 

c) SQLCODE 

Task 5 (Both) r - 

You want to use an INSTEAD OF trigger in your code. Find an'^xample that shows 
the correct syntax^ 

Task 6 (Both) 

You have written a PL/SQL program that dies because of a raised exception. Find 
an example of a PL/SQL exception handler. 

Task 7 (Prototype only) 

What causes ORA "twenty-forty" and what should you do to recover? 

Task 8 (Prototype only) 

You have never used two-phase commit. What things might go wrong with it? 

Task 9 (Both) 

Some of your queries are running slowly. A support person suggests that you set 
the ALWAYS_ANTI_JOIN parameter in init.ora to use the "sort merge algorithm". 
What is the exact syntax to use? 

Task 10 (Both) 

A query using a cursor is running slowly. You want to examine it using the catalog 
view V$SQL. What is the column in this view that contains the text of the SQL 
statement for the cursor? 
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Task 11 (Prototype only) 

You have been asked to give a presentation to top managennent on your plans to 
tal<;e advantage of LDAP and CORBA witliin OracleSi. You expect to do a fev\/ days 
of reading. 

a) Find a large body of information on LDAP. 

b) Since you will have to explain what the acronym mean, find what LDAP stands 
for. 

c) Find a large body of information on CORBA. 

d) Since you will have to explain what the acronym mean, find what CORBA 
stands for. 

Task 12 (Prototype only) 

While analyzing a large volume of sales data, your queries are slow. A support 
person suggests that you optimize your star queries. What steps should you take to 
do so? 

Task 13 (Prototype only) 

You cannot recover your database because all copies of the current control file have 
been destroyed by a media failure. What are the alternatives to recreate it? 

Task 14 (Prototype only) 

You are getting an error while calling a stored procedure. A support person asks 
you to send the DESCRIBE output for the procedure. What items will this output 
identify about the procedure? 

Task 15 (Prototype only) 

Under what circumstance would you use the ABORT option to shut down the 
database? 

Task 16 (Prototype only) 

Start at the "compact" list of books. Assume that your database or application is 
running too slowly. Find the information that will help you. 

Task 17 (Prototype only) 

Start at the "compact" list of books. Find information about the new features in the 
latest release. 

Task 18 (Prototype only) 

If you have not already tried the navigation paths, try each and comment on its 
usefulness/intuitiveness: 

a) List of books (summary view) 

b) List of books (tree view) 

c) Virtual book ("blue" search) 

d) Master glossary 

e) Master index. (Were you aware that this existed in the current documentation 
set?) 
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Participants 



Round 1: 



Name 


Jlob 


Company 


Doc. Library Usage 


David 


Sr. Product Mgr. 


Centura Software 


4 times per month 


Praveen 


Sr. Consultant/Oracle DBA 


eTouch Systems 


3-4 times per day 


Rajiv 


Lead DBA 


GAP Inc. 


4 times per month 


Edmond 


Software Developer 


Agilent Technologies 


1 time per week 


Mike 


Internet Application 

Architect 


SunGuard Treasury 
Systems 


6 times per year 


Ronald 


CTO 


Financial Engines 


1 time per month 


Jofin 


Prod. Mgr., Data 
Warehousing 


Oracle 


1 time per week 


John 


IT Mgr. 


Creative Leisure 


5 times per month 


Robin 


Owner/Network Svcs. 
Consultant 


Silicon Mirror 


None 


Tim 


SR. QA Engineer 


Digital Insight Corp. 


3 times per week 


Alex 


DBA 


ReleaseNow.com 


1 time per day 



Name 


Job 


Company 


Doc. Library Usage 


Barry 


Sr. Staff, Schwab 
Technology 


Charles Schwab 


1 time per month 


Andrea 


Sr. Tech. Staff, Web 
Development 


Oracle 


1 time per week 


Allen 


DBA 


Hewlett-Packard 


2 times per day 


Can 


Oracle DBA 


iPass Inc. 


2 times per week 


Brent 


Sr. Software Engineer 


Space Systems/Loral 


2 times per week 


William 


Database Project Lead 


U.S. Postal Service 


6 times per year 


Alan 


Director, Software 
Engineering 


Interlinear 
Technology Inc. 


1 time per month 


Hua 


DBA 


Symantec Corp. 


None 


Areiel 


Managing Principal 


Data Movement 
Research 


2 times per week 


Yolanda 


Principal Consultant 


Data Movement 
Research 


10 times per month 


Anu 


Software Engineer 


Hewlett-Packard 


2 time per week 
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Prototype Enhancements for Round 2 

For the virtual book: 

• Added an explanatory lead-in on the main page. 

• Made the form more compact, to avoid scrolling and make it easier to see results. 

• Allow pre-selected topics to be chosen from a list. 

• Added an index on the results page, with entries drawn from multiple books. 

• Added a bibliography to the results page. 

• On the results page, added links to PDF for all books that contribute to the virtual 
book, so that users print individual topics, chapters, or books for offline reading. 

For the tree-view search: 

• Added an explanatory lead-in on the main page. 

• Made the form more compact, to avoid scrolling and make it easier to see results. 

• On the results page, added the total number of hits, and how many books they 
come from. 

• On the results page, added an instruction to scroll if necessary to see results. 

• If there are more than 300 results, added a hint to try the virtual book or use the 
controls to filter by book or category. 

• Linked book titles to TOCs for corresponding books; users must click on "+" icons 
to expand. 

For the list of books: 

• Removed the tree view of books, but left the compact list of books. 

• Moved the link to list of books near the front of the main page. 

• Allowed users to select a category from a dropdown list, after which icons appear 
to indicate which books are in that category. 

For the main page: 

• Everything was visible without scrolling. 

• The Home, Feedback, and Help links were made more visible. 

• Introductions were added to each search method. 

• The main keywords were highlighted in red. 

For the master index: 

• Shortcuts were added. 

• The ability to search the index was added. 
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General Results 

Round 1 

• Users loved the virtual book feature, although few recognized or used it without 
prompting. 

• Most users were attracted to the tree search, and then stuck with it for all tasks. 

• Users complained that they did not recognize the prototype's array of navigation 
tools. 

• When directed to it, most users liked the organization and functionality of the 
"compact" list of books. 

• Mean ratings for the subset of six tasks performed using both methods (where 

1=Very Hard and 7=Very Easy) were: 

Documentation Library = 4.1 
Document Navigation Prototype = 5.3 

• The mean overall post-session rating for the Prototype was 5.9 (where 1 =Very Hard 
and 7=Very Easy). 

• Several users remarked that their ratings would have been even higher had they 
discovered the Prototype's full functionality sooner. 

• The mean post-session rating directly comparing the two methods was 6.3 (where 
1=Library Much Better, 7=Prototype Much Better). 

• The percentage of tasks successfully completed (subset of six tasks performed 
using both methods) was: 

Documentation Library = 82% (27 of 33) 

Document Navigation Prototype = 100% (28 of 28) 

• The mean time per task (subset of six tasks performed using both methods) was: 

Documentation Library = 3 minutes, 36 seconds (15% longer) 
Document Navigation Prototype = 3 minutes, 7 seconds 

Round 2 

• Users again liked the virtual book feature. In Round 2, more recognized it without 
prompting. 

• Most users tended to stick with one search until it failed. 

• More users noticed the prototype's array of navigation tools than in Round 1 . 
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• Mean ratings for the subset of six tasks performed using botli methods (where 

1=Very Hard and 7=Very Easy) were: 

Documentation Library = 4.0 
Document Navigation Prototype = 5.4 

• The mean overall post-session rating for the Prototype was 6.0 (where 1 =Very Hard 
and 7=Very Easy). 

• The mean post-session rating directly comparing the two methods was 6.4 (where 
1 =Library Much Better, 7=Prototype Much Better). 

• The percentage of tasks successfully completed (subset of six tasks performed 
using both methods) was: 

Documentation Library = 82% (27 of 33) 

Document Navigation Prototype = 97% (32 of 33) 

• The mean time per task (subset of six tasks performed using both methods) was: 

Documentation Library = 4 minutes, 37 seconds (28% longer) 
Document Navigation Prototype = 3 minutes, 37 seconds 

Usability Issues and Recommendations 

Round 1 

Issue 1.1 (Impact = High): 

Users did not discover the full range of navigational aids in the prototype. It took 
many users several tasks before they even noticed the "frequently accessed" 
hyperlinks. 

Re commendations: 

Use fewer interaction modes. Group the search tools into four types, each of which 
could occupy one clearly titled quadrant of the Home Page: 

• Frequent Types (4 hyperlinks, e.g., SQL and PL/SQL syntax and 
commands ) 

• Frequent Tools (4 hyperlinks, e.g.. Master Index ) 

• Keyword Search 

• Virtual Book 

Above the four quadrants, add the simple instruction "Use one of these four search 
methods". 

Consider numbering the quadrant titles 1 through 4. 
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Issue 1.2 (Impact = High): 

Most users had to be directly prompted to use the prototype's virtual book. 

Recommendations: 

The four-quadrant layout recommended above should draw more attention to the 
virtual book. 

Consider adding a brief description below the title of each quadrant. 

Provide help for each search tool. 

Issue 1.3 (Impact = High): 

Several users never scrolled to see the search tools located below the prototype's 

"tree search". 

Re commenda tion: 

Make all search functionality visible without scrolling (see recommendation for Issue 
1)- 

Issue 1.4 (Impact - High): 

Many used the prototype's tree search exclusively throughout the test session, 
although most did not notice the radio button or checkbox modifiers associated with 
the tree search. 

Recommendations: 

Giving equal visual "weight" to the four navigation "quadrants" may keep users from 
gravitating to the most familiar tool. 

Put the following GUI tools below the Keyword text box and "Search" button: 

• "Search In" list box [default = All books] 

• "Search" list box [default = Search all topics within selected book(s)] 

• "Search only text headers" checkbox 

• "Expand all results" checkbox 

Issue 1.5 (Impact = High): 

When using the Documentation Library, users were very annoyed when they taken 
to the top of a document rather than to the specific topic they had selected. 

Recommendations: 

Navigate directly to the first instance of the topic. 

Consider adding a tool (like the browser's "Find" function) that would allow users to 
find subsequent instances in the selected document. 

At least highlight each instance in the document. 
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Issue 1.6 (Impact = Medium): 

Most users were confused by the display of the search fields on the tree search 
results page. Since it took up so much room at the top of the page, several users 
invoked a repeat search (or a repeat virtual book) without ever looking at the 
results. 

Recommendations: 

Users are not used to seeing search results in the tested format. 

Restrict the page title to "Search Result". 

Below the title, "Your search found a total of X matching topics in Y document 
sections". 

Consider whether "sections", "books", or "volumes" is the appropriate term for the 
top level of the results tree. 

Label the (now more compact) search fields "Modify your search" or remove them 
entirely and offer a "Modify Search" button that returns users to the Home Page. 

Issue 1.7 (Impact = Medium): 

On the first day of testing, most users did not recognize the number shown in 
parentheses after each search document as the number of matching topics within 
that document. 

Recommendation: 

There was no confusion on the second day of testing when "X matching topics" was 
displayed. 

No further change recommended, except to Improve legibility by restricting the 
underlined hyperlink to the name of the document itself. 

Issue 1.8 (Impact - Medium): 

Most users reacted negatively to the 2- or 3-character "Shortcuts", especially when 
the shortcut table got very large. 

Recommendation: 

Replace the tree with a button that invokes the browser's "Find" dialog. 

Issue 1.9 (Impact = Medium): 

When users saw a virtual book for the first time, there was some confusion about 
the structure of the book. 

Recommendations: 

Make it immediately obvious that all books have four sections by labeling the jump- 
link section "This book has four sections". 

Make the jump links easier to read by underlining only the significant words 
(Background Information . How-to's . Examples , and Detailed Reference Info ). 
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Issue 1.10 (Impact - Low): 

Few users ever noticed the Back to home page link at the upper right of the 
prototype's second-level pages. 

Recommendation: 

Abbreviate the three upper-right hyperlinks (Home . Feedback . Help ) and use a 
larger, bold font. 

Include these links on all prototype pages. 
Round 2 

Issue 2.1 (Impact = High): 

Despite design changes for Round 2, a few users still did not notice the full array of 
choices on the prototype's home page. E.g., several users never noticed the 
"frequently accessed" links that exactly matched the type of information in tasks. 

Recommendations: 

Above the four quadrants, add the simple instruction "Use one of these four 
searcli metliods". 

Number and shorten the description of each quadrant: 

1. Frequently used links 

2. Master Index 

3. Keyword Search (Note: It's full-text by default only!) 

4. Virtual Book 

Add a Search Tips hyperlink next to the above instruction. 
Consider numbering the quadrant titles 1 through 4. 

Consider presenting the Master Glossary just like, and immediately under, the 
Master Index. Also, use a two-level shortcut list similar to that of the Master Index. 

Reduce distractions from the overall "wordiness" of the field labels and links. 

• For both Search and Virtual Book, replace "Enter a word or phrase. . ." with 
"Keyword(s)". 

• Replace "Display topics" with "Display". 

Replace "Look up an error message " with " Error Messages ". 

Replace "List of books " with " OracleSi Books (complete list)". 

Replace "Find a short definition in the Master Glossary " with " Master Glossary 
(definitions)". 

When searches return no results, add another tip that says "Return Home and try a 
different search method." 
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Issue 2.2 (Impact = High): 

Several users did not understand the relationship between the keyword field and 
the "pre-selected" list in the Virtual Book tool. 

Recommendations: 

Left-align the text entry field and the drop-down list. 

Put a boldface, capitalized "OR" between the entry field and the drop-down list (also 
left-aligned). 

Use the field labels "Keyword(s)" and "Select". 

Left-align the "Generate Book" button with the text field and list. 

Issue 2.3 (Impact = High): 

Few users ever noticed the checkboxes associated with the full-text search. 
Recommendation: 

Replace each checkbox with a pair of radio buttons. 

• © Search entire text O Search titles only 

• © Collapse results O Expand results 

Left-align the radio buttons with the keyword field and the drop-down lists. 

Issue 2.4 (Impact = High): 

Users continued to object when navigation took them to the start of a chapter, 
rather than to the first instance of the topic. 

Recommendations: 

Navigate directly to the first instance of the topic. 

Highlight each instance in the document. 

Issue 2.5 (Impact = Medium): 

Early in Round 2, users did not know to click on the icon to expand book titles into 
topics. 

Recommendations: 

Better integrate the "expand" (+) and "collapse" (-) icons into the book title. 

Continue to provide a separate Table of Contents hyperlink for each book. 

Change the instruction to say "Click a book title to view its matching topics." 
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Issue 2.6 (Impact - Medium): 

Users at first did not understand the structure produced by a Virtual Bool<. 

Recommendation: 

Replace the long explanation with a boldfaced header and list the following 
boldfaced contents, including number of matches, and using and avoiding 
hyperlinks as follows: 

Virtual Book Contents 

• Background Information (6 matching topics across 4 books) 

• Related Tasks (2 matching topics across 2 books) 

• Examples (2 matching topics in 1 book) 

• Detailed Reference information (1 matching topic) 

• Index Entries (12 matching topics across 6 books) 

• PDF files for printing (No matching topics) 

When a content section is empty, do not underline the title, and do add the 
following explanation: 

Issue 2.7 (Impact = Low): 

Some virtual book shortcuts are useless (e.g., when ALL matches start with the 
same initial characters. 

Recommendations: 

Build in some intelligence so that useless shortcuts do not take up valuable screen 
real estate. 

Issue 2.8 (Impact = Low): 

Like in Round 1 , some users complain that large shortcut tables are "low-tech". 
Recommen da tion : 

In a later release, allow users to turn the shortcut display on and off. 

Consider using only single-character or two-level shortcuts. 

Issue 2.9 (Impact = Low): 

Like in Round 1 , users did not notice the Home hyperlink. 
Recommendation: 

Double the separation between the three upper-right hyperlinks ( Home . Feedback . 
Help ) and remove the gray background to make them more legible. 
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Issue 2. 10 (Impact = Low): 

Most users liked the complete "list of books", although few perceived any use for 
the Figure List and Preface hyperlinl<s. 

Recommendation: 

Even if none of the other prototype tools are implemented, at least replace the 
current "Document Library" home page with the (much more complete and flexible) 
"list of books". 

Issue 2.11 (Impact = Low): 

Some users never use the browser's "Find" tool. 
Recommendation: 

Include a "Search Tips" link on all pages, and include "Find" as one tip. 
Issue 2.12 (Impact = Low): 

Most users never noticed the interMedia Text link, even when told that a keyword 
search was available for the Document Library. 

Recommendation: 

Integrate the search link into the page-top instructions as follows: 

Select a specific document or 

New! Search 8.1.6 documentation with interMedia Text 
Issue 13 (Impact = Low): 

Users had the most difficulty with Task 1 (find the command to set a default role for 

a user). 

Recommendation: 

On full-text search results pages, include search tips that recommends expanding 
books with the most matching topics. 

Issue 2.14 (No Change Recommended): 

Users understood and used the two-level shortcuts in the Master Index. 
Recommendations: 

Retain the latest design of the Master Index, including 

• Two-level "Shortcuts to Letters". 

• Providing full books names for each entry. 

• Identifying the multiple entries per book on separate rows. 

• Using an icon (>) to identify the first entry per book. 
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Issue 2.15 (No Change Recommended): 

In Round 2, users recognized the "X matching topics" number shown in 
parentheses after each book in the search results. 

Recommendation: 

Continue to use this format. 
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Results by Task 

Task 1 (Library & Prototype): 

You want to set a default role for a user. What command do you use? 



Results - Round 1 





Successful? 


Time 


Rating 


User 


Library 


Prototype 


Library 


Prototype 


Library 


Prototype 


1 


Y 




05:01 




3 




2 


Y 




03:02 




6 




3 




Y 




01:39 




5 


4 


N 




07:33 




1 




5 




Y 




04:56 




5 


6 


Y 




04:42 




4 




8 


Y 




02:07 




5 




9 




Y 




06:26 




3 


10 


N 




08:42 




2 




11 




Y 




03:57 




6 


Total 


4Y, 2N 


4Y 










Mean 






04:21 


04:14 


3.5 


4.8 


Results - 


Round 2 














Successful? 


Time 


Rating 


User 


Library 


Prototype 


Library 


Prototype 


Library 


Prototype 


1 


N 




04:51 




1 




2 




Y 




00:56 




6 


3 


N 




07:40 




1 




4 




Y 




06:12 




3 


5 


N 




09:04 




3 




6 




Y 




08:05 




2 


7 


Y 




04:29 




3 




8 




Y 




09:18 




5 


9 


Y 




01:00 




7 




10 




N 




11:09 




3 


11 


N 




04:23 




4 




Total 


2Y, 4N 


4Y, IN 










Mean 






05:14 


07:08 


3.2 


3.8 
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Task 2 (Library & Prototype): 

What is the difference between PCTFREE and PCTUSED? 



Results - Round 1 





Successful? 


Time 


Rating 


User 


Library 


Prototype 


Library 


Prototype 


Library 


Prototype 


1 


Y 




01:35 




4 




2 




Y 




04:56 




5 


3 


Y 




00:37 




7 




4 




Y 




01:12 




7 


5 


Y 




02:48 




5 




7 


Y 




03:03 




6 




8 




Y 




00:57 




6 


9 


Y 




04:30 




5 




10 




Y 




02:07 




6 


11 


Y 




01:01 




7 




Total 


6Y 


4Y 










Mean 






02:16 


02:18 


5.7 


6.0 


Results - 


Round 2 














Successful? 


Time 


Rating 


User 


Library 


Prototype 


Library 


Prototype 


Library 


Prototype 


1 




Y 




02:01 




7 


2 


Y 




03:47 




5 




3 




Y 




05:18 




6 


4 


Y 




01:06 




5 




5 




Y 




04:10 




6 


6 


Y 




04:57 




4 




7 




Y 




01:55 




6 


8 


Y 




09:05 




5 




9 




Y 




01:58 




6 


10 


Y 




03:08 




5 




11 




Y 




01:12 




6 


Total 


5Y 


6Y 










Mean 






04:25 


02:46 


4.8 


6.2 



Gary Felfoldy - User Focus, Inc. 



Oracle Document Navigation prototype Usability Testing - Final Report 



Task 3 (Prototype Only): 

What is the default length for a CHAR variable? What is the maximum length? 



Results 





Successful? 


Time 


Rating 


User 


Proto 1 


Proto 2 


Proto 1 


Proto 2 


Proto 1 


Proto 2 


1 




Y 




05:46 




4 


2 


Y 


Y 


01:28 


02:17 


5 


4 


3 


Y 


Y 


00:47 


04:39 


7 


5 


4 


Y 


Y 


01:34 


01:57 


7 


6 


5 


Y 


Y 


02:46 


03:00 


6 


5 


6 


Y 


Y 


06:10 


07:49 


4 


4 


7 


Y 


Y 


01:18 


05:51 


7 


4 


8 


Y 


Y 


00:41 


01:21 


7 


7 


9 


Y 


Y 


01:42 


00:27 


6 


7 


10 


Y 


Y 


00:56 


04:49 


7 


4 


11 


Y 


Y 


03:19 


00:56 


5 


7 


Total 


10Y 


11 Y 










Mean 






i O2:0|4 


03:32 


6.1 


5.2 
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Task 4 (Prototype Only): 

You want to use the following functions in your code. Find an example of each that 
shows the correct syntax. 

a) ADD_MONTHS 

b) DBMS_LOB.TRIM 

c) SQLCODE 



Results 





Successful? 


Time 


Rating 


1 — 

User 


Pro to 1 


Proto 2 


Proto 1 


Proto 2 


Proto 1 


Proto 2 






N 










2 


N 


Y 


04:13 


06:09 


4 


5 


3 


N 


Y 


04:03 


07:03 


6 


7 


4 


Y 


Y 


09:25 


07:31 


6 


6 


5 


Y 


N 


09:50 


15:35 


5 


5 


6 


Y 


Y 


06:02 


08:28 


4 


6 


7 


N 


Y 


11:03 


05:50 


5 


6 


8 


Y 


Y 


07:47 


04:22 


5 


7 


9 


Y 


Y 


14:17 


01:19 


5 


7 


10 


N 


N 


7:56 


07:24 


4 


6 


11 


Y 


N 


7:34 


08:20 


6 


5 


Total 


6Y, 4N 


7Y, 4N 










Mean 






08:13 


07:38 


5.0 


5.9 
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Task 5 {Library & Prototype): 

You want to use an INSTEAD OF trigger in your code. Find an example tliat shows 
the correct syntax. 



Results - Round 1 





Successful? 


Time 


Rating 


User 


Library 


Prototype 


Library 


Prototype 


Library 


Prototype 


1 


Y 




01:19 




4 




2 


Y 




02:10 




7 




3 




Y 




00:44 




7 


4 


Y 




05:05 




6 




7 




Y 




03:51 




7 


8 


Y 




05:08 




2 




9 




Y 




02:35 




5 


10 


N 


Y 


05:20 


00:36 


1 


7 


11 




Y 




00:49 




6 


Total 


4Y, 1N 


5Y 










Mean 






03:36 


01:43 


4.0 


6.4 


Results - 


Round 2 














Successful? 


Time 


Rating 


User 


Library 


Prototype 


Library 


Prototype 


Library 


Prototype 


1 


Y 




07:37 




4 




2 




Y 




01:29 




6 


3 


Y 




06:56 




5 




4 




Y 




00:50 




7 


5 


Y 




03:42 




5 




6 




Y 




01:22 




6 


7 


Y 




04:35 




5 




8 




Y 




12:01 




4 


9 


Y 




00:51 




7 




10 




Y 




01:16 




7 


11 


Y 




05:24 




3 




Total 


6Y 


5Y 










Mean 






04:51 


03:24 


4.8 


6.0 
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Task 6 (Library & Prototype): 

You have written a PL/SQL program that dies because of a raised exception. Find 
an example of a PL/SQL exception handler. 



Results - Round 1 





Successful? 


Time 


Rating 


User 


Library 


Prototype 


Library 


Prototype 


Library 


Prototype 


1 


Y 




03:07 




5 




2 




Y 




02:01 




4 


3 


Y 




01:23 




6 




4 




Y 




02:19 




7 


5 


Y 




01:51 




6 




6 




Y 




02:07 




4 


7 


Y 




05:28 




1 




8 




Y 




02:30 




3 


9 


Y 




06:10 




4 




10 




Y 




03:21 




5 


11 


Y 




01:25 




7 




Total 


6Y 


5Y 










IVIean 






03:14 


02:28 


4.8 


4.6 


Results - 


Round 2 














Successful? 


Time 


Rating 


User 


Library 


Prototype 


Library 


Prototype 


Library 


Prototype 


1 




Y 




07:03 




4 


2 


Y 




02:57 




3 




3 




Y 




03:43 




6 


4 


Y 




01:00 




5 




5 




Y 




01:30 




5 


6 


Y 




02:49 




3 




7 




Y 




00:39 




7 


8 


Y 




02:05 




7 




9 




Y 




01:16 




7 


10 


Y 




06:00 




2 




11 




Y 




01:09 




7 


Total 


5Y 


6Y 










Mean 






02:58 


02:33 


4.0 


6.0 
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Task 7 (Prototype Only): 

What causes ORA "twenty-forty" and what should you do to recover? 

Results 

There were too few data to analyze (only one observation per prototype). 

Task 8 (Prototype Only): 

You have never used two-phase conrimit. What things might go wrong with it? 

Results 

There were too few data to analyze (only one observation per prototype). 
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Task 9 (Library & Prototype): 

Some of your queries are running slowly. A support person suggests that you set 
the ALWAYS_ANTI_JOIN parameter in init.ora to use the "sort merge algorithm". 
What is the exact syntax to use? 



Results - Round 1 





Successful? 


Time 


Rating 


User 


Library 


Prototype 


Library 


Prototype 


Library 


Prototype 


1 


Y 




03:58 




4 




2 


Y 




00:52 




6 




3 




Y 




06:58 




4 


4 


Y 




04:32 




6 




5 




Y 




00:19 




7 


6 


N 




09:28 




3 




7 




Y 




06:02 




3 


8 


Y 




01:11 




6 




9 




Y 




06:45 




3 


10 


Y 




05:10 




4 




11 




Y 




01:26 




7 


Total 


5Y, IN 


5Y 










Mean 






04:12 


04:18 


4.8 


4.8 


Results - 


Round 2 














Successful? 


Time 


Rating 


User 


Library 


Prototype 


Library 


Prototype 


Library 


Prototype 


1 


Y 




03:53 




5 




2 




Y 




02:22 




6 


3 


N 




09:21 




2 




4 




Y 




00:56 




5 


5 


Y 




06:29 




2 




6 




Y 




01:01 




6 


7 


Y 




04:32 




4 




8 




Y 




05:20 




5 


9 


Y 




02:34 




5 




10 




Y 




08:00 




3 


11 


Y 




08:17 




3 




Total 


5Y, IN 


5Y 










Mean 






05:51 


03:32 


3.5 


5.0 
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Task 10 (Library & Prototype): 

A query using a cursor is running slowly. You want to examine it using the catalog 
view V$SQL. What is the column in this view that contains the text of the SQL 
statement for the cursor? 



Results - Round 1 





Successful? 


Time 


Rating 


User 


Library 


Prototype 


Library 


Prototype 


Library 


Prototype 


1 


Y 




02:06 




1 




2 




Y 




02:10 




6 


3 


N 




06:48 




1 




4 




Y 




07:15 




5 


7 


Y 




02:55 




4 




8 




Y 




05:38 




2 


9 


N 




04:45 




1 




10 




Y 




01:48 




6 


11 




Y 




01:40 




6 


Total 


2Y, 2N 


5Y 










Mean 






04:08 


03:07 


1.8 


5.0 


Results - 


Round 2 














Successful? 


Time 


Rating 


User 


Library 


Prototype 


Library 


Prototype 


Library 


Prototype 


1 




Y 




00:58 




7 


2 


Y 




00:49 




6 




3 




Y 




01:36 




5 


4 


Y 




01:32 




5 




5 




Y 




02:53 




6 


6 


Y 




05:51 




2 




7 




Y 




07:30 




5 


8 


N 




06:29 




1 




9 




Y 




00:50 




7 


10 


Y 




04:56 




4 




11 




Y 




03:11 




3 


Total 


4Y, IN 


6Y 










Mean 






03:55 


01:50 


3.6 


5.5 
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Task 11 (Prototype Only): 

You have been asked to give a presentation to top management on your plans to 
take advantage of LDAP and CORBA within OracleSi. You expect to do a few days 
of reading. 

a) Find a large body of information on LDAP. 

b) Since you will have to explain what the acronym mean, find what LDAP stands 

for. 

c) Find a large body of information on CORBA. 

d) Since you will have to explain what the acronym mean, find what CORBA 
stands for. 



Results 





Successful? 


Time 


Rating 


User 


Proto 1 


Proto 2 


Proto 1 


Proto 2 


Proto 1 


Proto 2 


2 




Y 




06:16 




6 


4 




Y 




11:44 




7 


5 


Y 


Y 


12:21 


05:58 


7 


7 


6 




Y 




04:42 




6 


8 


Y 




06:42 




5 




9 




Y 




01:55 




7 


10 


Y 


Y 


04:21 


07:39 


5 


6 


11 




Y 




08:45 




7 


Total 


3Y 


7Y 










Mean 






07:48 


06:43 


5,7 


6.6 
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Task 12 (Prototype Only): 

While analyzing a large volume of sales data, your queries are slow. A support 
person suggests that you optimize your star queries. What steps should you take to 
do so? 



Results 





Successful? 


Time 


Rating 


User 


Proto 1 


Proto 2 


Proto 1 


Proto 2 


Proto 1 


Proto 2 


1 




N 




07:31 




1 


2 


Y 


Y 


11:38 


01:16 


4 


5 


3 


Y 




03:32 




5 




4 


Y 




04:25 




7 




7 


Y 




12:13 




2 




8 


Y 




01:29 




4 




9 


Y 




01:55 




6 




10 


Y 




04:28 




4 




Total 


7Y 


1Y, 1N 










Mean 






05:40 


04:24 


4.6 


3.0 
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Task 13 (Prototype Only): 

You cannot recover your database because all copies of the current control file have 
been destroyed by a nnedia failure. What are the alternatives to recreate it? 



Results 





Successful? 


Tinne 


Rating 


User 


Proto 1 


Proto 2 


Proto 1 


Proto 2 


Proto 1 


Proto 2 


1 




N 




06:05 




5 


2 


Y 


N 


02 


20 


05:34 


6 


2 


8 


Y 




01 


43 




6 




9 


Y 




01 


12 




7 




10 


Y 




06 


49 




6 




Total 


4Y 


2N 










Mean 






03:01 


05:50 


6.3 


3.5 
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Task 14 (Prototype Only): 

You are getting an error while calling a stored procedure. A support person asks 
you to send the DESCRIBE output for the procedure. What itenns will this output 
identify about the procedure? 



Results 





Successful? 


Time 


Rating 


User 


Proto 1 


Proto 2 


Proto 1 


Proto 2 


Proto 1 


Proto 2 


1 




N 




05:54 




1 


2 


Y 


Y 


02:30 


05:41 


6 


6 


3 


N 


N 


10:25 


19:05 


1 


5 


4 


Y 


Y 


06:11 


05:37 


3 


4 


5 




Y 




02:33 




7 


6 




Y 




04:05 




5 


7 


Y 


N 


01:56 


16:33 


7 


3 


8 


N 


N 


11:48 


09:43 


2 


2 


9 


Y 


Y 


02:36 


00:38 


7 


7 


10 


Y 


Y 


04:25 


03:45 


4 


6 


11 


N 


Y 


13:55 


02:47 


2 


4 


Total 


5Y, 3N 


7Y,4N 










Mean 






06:37 


06:56 


4.0 


4.5 
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Task 15 (Prototype Only): 

Under what circumstance would you use the ABORT option to shut down the 
database? 



Results 





Successful? 


Time 


Rating 


User 


Proto 1 


Proto 2 


Proto 1 


Proto 2 


Proto 1 


Proto 2 


1 




Y 




00:54 




7 


2 


N 


Y 


02:40 


01:24 


6 


6 


3 


Y 


N 


00:54 


03:04 


7 


1 


4 


Y 


Y 


03:03 


01:32 


7 


6 


5 




Y 




02:01 




7 


6 




Y 




02:05 




5 


7 


Y 


Y 


07:54 


01:45 


5 


6 


8 


Y 


Y 


03:23 


02:58 


3 


7 


9 


Y 


Y 


01:12 


00:22 


6 


7 


10 


Y 


Y 


00:39 


01:18 


7 


7 


11 


Y 


Y 


01:01 


02:05 


7 


7 


Total 


7Y, 1N 


10Y, 1N 










Mean 






02:36 


01:46 


6.0 


6.0 
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Task 16 (Prototype Only): 

Start at the "compact" list of books. Assume that your database or application Is 
running too slowly. Find the information that will help you. 

Results 

There were too few data to analyze (fewer than two observations per prototype). 



Task 17 (Prototype Only): 

Start at the "compact" list of books. Find information about the new features in the 
latest release. 

Results 

There were too few data to analyze (fewer than two observations per prototype). 



Task 18 (Prototype Only): 

If you have not already tried the navigation paths, try each and comment on its 
usefulness/intuitiveness: 

a) List of books (summary view) 

b) List of books (tree view) 

c) Virtual book ("blue" search) 

d) Master glossary 

e) Master index. (Were you aware that this existed in the current documentation 
set?) 

Results 

There were too few data to analyze (fewer than two observations per prototype). 
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Appendix 



statement of Informed Consent 
Task Instructions 
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Statement of Informed Consent 



Purpose: 



You have been asked to participate in a usability evaluation of Oracle's 
online Documentation, By participating in this evaluation, you will help us 
mal<e the information easier to navigate and use. This evaluation is 
designed to test the design of the tool; it is not intended to test your 
individual performance in any way. 



You will be asked to perform some typical tasks and answer some 
questions about your information needs. While you work, I will videotape 
your interactions and record information about how you use the tools. 
Aftenwards, I will talk with you about your impressions. If other team 
members stop by to observe from the Control Room, they may want to 
participate in the discussion as well. 



I will use the data you give us, along with the information I collect from 
other participants, to make recommendations aimed at improving users' 
experience with the documentation. To ensure confidentiality, I will not 
associate your name with your data. 



We will not have any scheduled breaks. However, you may take a break 
at any time. 

Procedure to Withdraw: 

You may withdraw from the evaluation at any time. 



If you agree to these terms, please indicate your acceptance by signing below. 

Signature: 

Printed Name: 
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Procedure: 



Confidentiality: 
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Task Instructions 
Procedure: 

• Each task is described on a separate page. 

• I will read the task aloud. 

• Ask any questions you have 

• Indicate when you are ready to begin 

• Think aloud as you perform the task 

• Describe step-by-step what you are doing, not just what you are thinking 

• Indicate when you have finished the task 

• Answer the questions at the bottonn of the page 

• Make comments & suggestions 
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